All commit authors signed the Contributor License Agreement.

@pablogsal can you take a look this week? 🙏

@pablogsal can you take a look this week? 🙏

I can try but I have some other PRs first in my review queue :(

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

Unfortunately the more I think about it the less I like it: this model is harder to reason about than the previous linked-list model, and I think that matters because it creates a very easy footgun: the current chunk looks like the current stack backing store, but it is not.

Existing live frames may still be in older chunks, while newer frames are in the newest chunk at a matching logical offset. So any code that treats stack_chunk_list as “the stack” instead of “the head of a chain that may contain live frames” can silently become wrong. That already seems to happen here: copying only the head chunk misses older live frames and turns what used to be a cheap bulk-copy unwind into many per-frame remote reads.

The risk is not just external profilers. This makes future runtime/debugger code more fragile because pointer validity and frame ownership now depend on searching the whole chunk chain, not checking the current chunk.

@pablogsal
I don't understand why this PR would cause 1000 remote reads. A 1000 frames stack will only span a few chunks, since they grow exponentially. For very large stacks, there will be fewer chunks to copy, not more.

Existing live frames may still be in older chunks, while newer frames are in the newest chunk at a matching logical offset.

There seems to be a misunderstanding here. No two frames will have the same offset.

This makes future runtime/debugger code more fragile because pointer validity and frame ownership now depend on searching the whole chunk chain, not checking the current chunk.

The stack is a linked list of frames, not chunks. Some of which (generator and coroutine frames) aren't in chunks at all, so tools already need to handle pointers outside of the current chunk.

Overall, I don't see how this really changes anything for an out-of-process profiler: Copy all the chunks, then traverse the stack.

Also, note that the lower, unused part of the current chunk in a stack with multiple chunks will be untouched, so a profiler should be able to detect when it needs to cross to another chunk.

Other profilers like austin currently hard-codes the old _PyStackChunk layout (previous, size, top, data), while this patch changes it to (size, previous, data), so those tools need explicit updates.

That is trade off that tools and libraries make: either they use stable API/ABIs, or they probe CPython internals. If they do the latter, they will need updating every release.

@P403n1x87 would this be a problem for you?

I don't understand why this PR would cause 1000 remote reads. A 1000 frames stack will only span a few chunks, since they grow exponentially. For very large stacks, there will be fewer chunks to copy, not more.

The issue I was pointing at is the current _remote_debugging implementation in this PR: copy_stack_chunks() now processes only the head chunk and sets out_chunks->count = 1. Once frame->previous points into an older chunk, find_frame_in_chunks() misses and we fall back to parse_frame_object(), which reads the frame remotely. That is where the ~966 remote reads for a 1000-frame stack came from.

The fix here is to either restore eager copying of the full stack_chunk_list -> previous chain using the new _PyStackChunk layout, or lazily copy previous chunks when the frame walk crosses out of the copied head chunk.

There seems to be a misunderstanding here. No two frames will have the same offset.

Also, you’re right that “same logical offset” was poor wording. I meant that the new chunk starts allocating at the old stack depth, leaving the lower part of the new chunk unused; not that two frames have the same offset.

Here is a repro:

import os
import subprocess
import sys
import _remote_debugging

DEPTH = 1000

child_code = f"""
import os, sys, time
sys.setrecursionlimit({DEPTH + 1000})

def f(n):
    if n == 0:
        print("READY", os.getpid(), flush=True)
        time.sleep(10)
        return
    f(n - 1)

f({DEPTH})
"""

p = subprocess.Popen(
    [sys.executable, "-c", child_code],
    stdout=subprocess.PIPE,
    text=True,
)

line = p.stdout.readline().strip()
pid = int(line.split()[1])

try:
    unwinder = _remote_debugging.RemoteUnwinder(
        pid,
        all_threads=True,
        cache_frames=False,
        stats=True,
        debug=True,
    )
    trace = unwinder.get_stack_trace()
    frames = sum(len(t.frame_info) for interp in trace for t in interp.threads)

    print("frames:", frames)
    print("stats:", unwinder.get_stats())
finally:
    p.terminate()
    p.wait()

In main we can see:

❯ ./python repro.py
frames: 1002
stats: {'total_samples': 1, 'frame_cache_hits': 0, 'frame_cache_misses': 0, 'frame_cache_partial_hits': 0, 'frames_read_from_cache': 0, 'frames_read_from_memory': 0, 'memory_reads': 4, 'memory_bytes_read': 1200, 'code_object_cache_hits': 1000, 'code_object_cache_misses': 2, 'stale_cache_invalidations': 0, 'frame_cache_hit_rate': 0.0, 'code_object_cache_hit_rate': 99.8003992015968}

with this PR:

frames: 1002
stats: {'total_samples': 1, 'frame_cache_hits': 0, 'frame_cache_misses': 0, 'frame_cache_partial_hits': 0, 'frames_read_from_cache': 0, 'frames_read_from_memory': 0, 'memory_reads': 966, 'memory_bytes_read': 85768, 'code_object_cache_hits': 1000, 'code_object_cache_misses': 2, 'stale_cache_invalidations': 0, 'frame_cache_hit_rate': 0.0, 'code_object_cache_hit_rate': 99.8003992015968}

Notice memory_reads going from 4 to 966

The fix here is to either restore eager copying of the full stack_chunk_list -> previous chain using the new _PyStackChunk layout, or lazily copy previous chunks when the frame walk crosses out of the copied head chunk.

Yes, that's my bad. I made the smallest changes possible to the remote debugging module to get the PR working, but didn't inspect further improvements. Maybe it can be done in a follow-up PR by people more knowledgeable on the module? Or would you consider that a blocker?

Maybe it can be done in a follow-up PR by people more knowledgeable on the module? Or would you consider that a blocker?

This is a blocker. This PR adds a regression and that's not acceptable.

@dpdani I pushed a fix for the concrete _remote_debugging regression: it now walks tstate->stack_chunk_list through previous and copies all stack chunks before traversing frames, instead of only copying the newest chunk. With the fix, the 1000-recursive-frame repro goes from roughly ~966 remote reads / ~85 KB read to 3 reads.

I also added a regression test to make sure deep stacks are resolved from copied chunks rather than falling back to parsing frames individually from remote memory.

That said, I still think this solution is very confusing and too complex. It is harder to reason about where frames live, which chunks are relevant, and what invariants the implementation can rely on. I am worried this makes future changes easier to get subtly wrong.

I investigated an alternative implementation in #149097: instead of replacing the stack with the resizable-array model, it keeps the current chunked- stack invariants and extends the existing one-chunk cache into a small bounded per-thread cache.

I think this is a better direction because it fixes the allocator-thrashing issue without changing where live frames can reside, without changing _PyStackChunk/PyThreadState layout, and without requiring _remote_debugging or external unwinders to learn a new stack model. Active frames remain only in the datastack_chunk -> previous chain; cached chunks are detached and inactive.

The cache is intentionally bounded: currently up to 8 * _PY_DATA_STACK_CHUNK_SIZE, so the memory cost is predictable and much smaller than retaining a high-water-mark stack for the lifetime of the thread.

I cehcked the original repro and it no longer shows per-branch mmap/munmap churn.